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Desirable Mail Questionnaire Characteristics 
in Teacher Education Research 

Researchers have commented that the nature of the population being surveyed should be taken 
into consideration when survey techniques are selected (Baumgartner & Heberlein, 1984; Berdie, 
Anderson, & Niebuhr, 1986; Jones, 1979; Mason, Dressel, & Bain, 1961; Suctman, 1985), and they have 
recommended caution in generalizing research results beyond the target population. More specifically, 
Sudman (1985) has posed the idea that effective survey procedures for professionals may not be 
identical to those used with samples from the general population. 

There are many dimensions to a survey. The population surveyed, the focus of the survey, and 
the number of attempts to contact nonrespondents are but a few of the many ways in which surveys may 
differ and influence the effectiveness of the survey. The questionnaire itself is one aspect (a very 
important one) of a mail survey. There is some evidence that questionnaire variations can be used with 
equal success. Nonprofessional questionnaire construction and style of type have been as effective in 
surveying college graduates as the professional-appearing instrument recommended by some authors 
(Boser, 1990). Similarly, placement in the survey of open-ended and demographic items did not 
influence overall response rate, item responses, or completeness of responses (Jensen & Busk, 1991). 

Previous research has sought to identify mail questionnaire characteristics that would be 
accepted by experts and general survey researchers as desirable (Boser & dark, 1990; Clark & Boser, 
1989). While the experts in the study did not unanimously agree on many of the characteristics, support 
from a second group of experienced survey researchers reduced even further the number of desirable 
characteristics (38 of the total 82 items) about which there was general agreement. Novice survey 
researchers tended to agree with the experts regarding questionnaire characteristics but were more 
conservative (Green, 1991). There exists the possibility, however, that some mail questionnaire 
characteristics, may be population- or study-specific rather than univei ally applicable. 

One particular type of survey that is conducted hundreds of times each year in different 
locations by different institutions is the follow-up survey of program graduates. Procedures used in 
these studies vary considerably (Boser, 1988). To aid those who conduct various types of surveys, Roth 
(1981) developed a step-by-step manual for conducting surveys. He included checklists on question 
construction and format based partially on previously published materials from the 195Cs. 

Other sources dealing with follow-up surveys have utilized a variety of approaches but 
generally contain little about questionnaire development. McKenna (1983) assembled sample 
questionnaires from follow-up studies, accompanied by reprints of five brief published articles. 
Freeman (1988) compiled individual items in a variety of formats from some existing questionnaires into 
a sourcebook of items. In a special journal issue devoted to alumni research, only one article (Fisher, 
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1988) addressed survey instruments for mail and telephone surveys, and that topic was covered in five 
paragraphs. 

The present study is an attempt to identify desirable mail questionnaire characteristics as 
perceived by those who conduct successful surveys (those with a high return rate) of teacher education 
program graduates. University sponsorship is almost always inherent in such studies. Some 
characteristics of the participants in this study have been controlled by limiting it to this particular 
type of survey effort. The members of the target populations for the respective surveys were familiar 
with the survey sponsor, potentially loyal to the sponsoring institution, and somewhat homogeneous in 
various respects (educational level, occupational objectives, etc.). For purposes of this study, data were 
sought from individuals regarding only their surveys of teacher education program graduates in their 
first year beyond program completion in an attempt to limit the extent to which response rate would be 
influenced by inability of the researcher to locate the graduates. 

Method 

Based on a previous survey of all teacher education programs that were accredited by the 
National Council for Accreditation of Teacher Education and members of the American Association of 
Colleges for Teacher Education (Boser, 1988), 178 programs that surveyed teacher education program 
graduates during the first year following program completion were identified. Difficulty in locating 
graduates can be expected to increase with the passage of time after leaving the institution. This study 
attempted to minimize variability due to that source by limit ng it to studies of first-year graduates. 
Instrument 

The survey instrument consisted of three parts. Each part was on a separate 8 1/2" by 14" sheet, 
folded in the middle with printing on both sides to provide four T by 8 1/2" pages. The outer sheet was 
ivory-colored, the middle sheet was salmon-colored, and the inner *heet was gray. Questions on the 
first part (the outer sheet) served to screen respondents to determine whether or not they fit the criteria 
for the study and to obtain demographic information. The second part of the instrument (salmon- 
colored) was titled Procedures Used in Most Recent Vollovntp Surveys and asked 25 factual questions 
about the most recent survey of first-year graduates of the teacher education program. The inner sheet, 
General Const derations in Design of Questionnaires, contained 82 items that were used in the Boser and 
Clark (1990) validation study. Respondents were asked to check in the blank beside each item that 
they considered to be important when they designed questionnaires for their respective follow-up 
surveys of graduates. 
Procedures 

In July of 1990, the individual identified as the contact person at each institution in the 
previous survey was sent the three-part questionnaire set and an explanatory cover letter. The 
recipient was requested to direct the materials to the person who had conducted the most recent 



follow-up survey of graduates, if that person were someone other man the recipient. One follow-up 
mailing was sent approximately one month later. Of the 113 (635%) returned questionnaires, 24 
contained information mat excluded mem from the survey. Of the remaining 89, four did not respond to 
any items on the General Considerations instrument, thus reducing the data pool to 85. 

Responses to items on the Procedure* Used in Most Recent Follow-up Survey were used to 
calculate response rate for that survey. Response rate was determined by subtracting the number of 
individuals who could not be located from the total number sent questionnaires, then dividing this 
number into the number who responded to the survey. Survey response rates could not be calculated 
accurately for five of the 85 in the data pool, leaving a total of 80. 

The calculated response rates varied from 15% to over 90%. The intent of this study was to 
identify desirable characteristics (those that would be associated with successful surveys), so the 
responses from the 25% of the individuals (n=20) with the highest response rates were selected for 
analysis. Response rates for this target group were from 69% to 91%. Eight of the respondents supplied 
a copy of the questionnaire used in the study about which they reported. 

Seven of the 20 conducted only follow-up surveys. Eighteen were employed in schools, colleges, 
or departments of education; the remaining two worked in career planning/ placement units. 
Participants had an average of 13.6 years experience in survey research (see Table 1). The size of the 
target populations ranged from 15 to 2698. 

The number of responses required for questionnaire completion also varied considerably. Most of 
the questionnaires (75%) contained two pages of questions or less. Only two of the instruments were 
constructed in booklet format (one with 86 questions and one with 52). 
Ana lysis 

Responses on the Central Considerations form for the target group were analyzed by calculating 
percentages of the 20 respondents checking each item as important when designing follow-up survey 
questionnaires. Perceived importance of questionnaire characteristics was determined by the 80% 
agreement level. Sample questionnaires submitted by participants in the survey and responses to items 
on the Procedures Used in Most Recent Follow-up Survey were examined to better understand the 
importance or lack thereof regarding specific questionnaire design considerations. 

Results and Discussion 

A list of all items by category and their respective response percentages are listed in the 
Appendix. Percentages of the 20 respondents considering the various characteristics to be important 
ranged from 5% to 100%. Using the 80% agreement that was used in the previous validation study 
(Boser & Clark, 1990), only 18 items were perceived as important for mail questionnaires used in follow- 
up surveys of graduates (see Table 2). Fourteen of those items were among the 38 that had been previous 
recommended by the panel of experts and validated by the group of survey practitioners. 



Item 


Response 


n 


% 


Years Experience Conducting Survey* 


3-5 


4 


20 




6-10 


6 


30 




11-15 


2 


10 




16-20 


5 


25 




21-25 




5 




26-30 


j 


5 




31-32 




5 


Students Sent Questionnaires 


15 


1 


5 




18 




5 




20 


j 


5 




26 




5 




29 




5 




30 




5 




37 


I 


5 




53 


1 


5 




75 




5 




115 


} 


5 




136 


i 


5 




150 




5 




200 


i 


5 




214 




5 




300 




5 




354 


i 


5 




360 


i 


5 




560 


i 


5 




600 




5 




2698 




5 


Maximum Number of Responses 








to Questionnaire 


3 




5 




7 




5 




9 




5 




10 


i 


5 




15 


i 


5 




19 


i 


5 




20 




10 




25 




10 




26 


i 


5 




30 




5 




35 


i 


5 




37 


i 


5 




38 




5 




40 




5 




52 


; 


5 




/53 




5 




71 




5 




86 




5 


Questionnaire Construction 


One page, one side 


2 


10 




Two pages 








One sheet front & back 


7 


35 




Two sheets* stapled 


6 


30 




Three pages* 


2 


10 




Four pages* 


2 


10 




Six pages 


1 


5 



•One in booklet format 

4 



Table 2 

Desirable Mall Questionnaire Characteristics 
Important for Follow-up Surveys of Program Graduates 

Percentage 
Rating Item 



Item Important 
Previously Validated Items 

A2. The instrument looks easy to complete. 100 

B2. The instructions are brief, 100 

A6. The type is clear ami legible. 95 

CI 1 . There is adequate space for responding, 95 

E3. All items arc essential ami relevant to the purposes of the survey. 95 

E2. Each item seeks just one piece of information, 90 

El. The respondent is able to provide answers to the questions in the 

instrument 85 

A5. Printing does not bleed through the paper. 80 

B6a. The instructions are clean They specify when to put a check mark 

and when to write in a response. 80 

C3 Each item and its response options are on the same page. 80 

Dlb. The initial items are applicable to all members of the survey population. 80 

Fl . The choice of words is appropriate to the literacy level of the 

survey population. 80 

F3d. Items are simple, direct, and unambiguous. They do not contain 

instances of double negatives in items and /or response options. 80 

Gig. Response options are appropriate for the item. 80 

Items Important for Follow-up Surveys But Not Previously Validated 

Bl . General instructions that apply to the entire instrument are provided at the 

beginning of the instrument 90 

Glf. Response options are brief. 90 

A9b. The front page (or cover) contains general directions. 85 

Dla. The initial items are clearly connected to the stated purpose of the survey. 85 



There were four additional items thought to be important in follow-up surveys of graduates 
that were given less weight in general surveys. Comments in the previous study (Boser & Clark, 1990) 
indicated that experts and other experienced survey researchers sometimes thought general directions 



(Bl and A9b) might be better placed in the cover letter than in the questionnaire. The follow-up 
researchers, however, appear to be more inclined to include the general instructions on the 
questionnaire, although one individual in the present study did note that these were in the cover letter. 
A possible explanation may lie in the nature of the cover letter (if a separate cover letter is included). 
Two of the eight sample questionnaires included a brief, one paragraph cover letter-type message to 
the graduate at the top of the first page of the questionnaire. A third questicnnaire had a similar brief 
message stapled to the top of the questionnaire. 

The follow-up researchers also thought response options should be bnsf (Glf) and initial items 
should be connected to the purpose of the survey (Dla). Both of these items were considered important 
by the original panel of experts but were not validated by the panel of experienced researchers in the 
previous study (Boser & Clark, 1990). 

By way of contrast, twenty-four items that were previously validated wet not considered 
important by this group (see Table 3). As previously mentioned, there are some fair v unique or 
distinguishing aspects of this type of endeavor. Institution 

sponsorship is almost always obvious, and there may be emotional ties between the graduates and the 
institution that facilitate return of questionnaires. The follow-up surveyors felt less need for politeness 
(B4, 75%) and expressing appreciation to the participants (All, 70%) than did the survey researchers 
and experts in the previous research (Boser & Clark, 1990). 

Follow-up researchers were more likely to place a value on making initial items 
nonthreatening (Did, 70%) and easy (Die, 65%) than interesting (Die, 35%). It may be that because of 
the nature of their task, the follow-up researchers did not perceive the items or the study title (Al, 
60%) were likely to be especially appealing or interesting to the participants. 

The follow-up researchers may also have felt it was less important to have adequate margins 
and avoid the appearance of crowding ( A3,65%) because they did not have to "sell" the participants on 
completing and returning the questionnaires. There was wide variation in margins among the eight 
sample questionnaires, with side margins ranging from 1/4 inch to 1 1/2 inches; top margins from none to 
1 inch; and bottom margins from 1/2 to 1 1/2 inches. The appearance of the questionnaire with respect to 
margins and crowding is largely a matter of judgment, and such considerations may have been sacrificed 
in the interest of keeping the apparent size of the instrument sma'i. All of the researchers thought it 
important that the instrument took easy to complete (A2, see Table 2). Fifteen of the 20 questionnaires 
consisted of no more than two pages of questions. 

Some of the lack of support for these items may stem from a basic flaw in the design of the 
current data collection instrument. Respondents were instructed to check beside items they considered 
important, but there was no code for marking items that were not applicable. Item CI 4b is related to 
checklists in which column headings are carried over from one page to another (35%). This situation 
did not occur in four of the eight sample questionnaires. On only one of the four remaining questionnaires 
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Table 3 



Previously Validated Items Not Considered Important 
in Follow-up Surveys erf Graduates 



Percentage 
Rating Item 



Item Important 

B4. The tone of the directions is polite (e.g., "please"). 75 

F3b. Items arc simple, direct and unambiguous. They do not contain instances of 

"loaded" items (that use emotionally colored words). 75 

F3e. Items are simple, direct, and unambiguous. They do not contain instances of 

negatively worded items coupled with agree/disagree response format. 75 

Al 1. Appreciation for completing the instrument is expressed. 70 

Did. The initial items are nonthreatening. 70 

A3. Margins are adequate: instrument doesn't look crowded. 65 

Die. The initial items are easy. 65 

F3g. Items are simple, direct, and unambiguous. They do not contain instances of 

"giveaway" words (e.g., "all"). 65 

Al. The title of the study/questionnaire is likely to appeal to the survey population. 60 

C9. Response options are close to the item stem. 60 

C8. Response options are arranged vertically (or in c jlumns if several consecutive 

items use the same response options)- 55 

D6. Items with similar content are grouped together; within each content group, items 

with the same response format are presented together. 45 

C7. When response options are provided (including, if appropriate, a response of 

"other"), each response options has either a numeric or alphabetic code beside it. 40 

Gla. Response options exhaust all possibilities or include "other," "undecided/' 

or "neutral" category. 40 

Clb. Response options are mutually exclusive, 40 

Gld. Response options do not contain more than one alternative that could be correct 

unless multiple responses are allowed, 40 

C13. When ranking, the number of items to be ranked is limited (e.g., three best and 

three worst). 35 

CI 4b. For checklists, column headings are carried over from one page to another. 35 

Die. The initial Items are interesting. 35 

E4b. For items used for skip/filter/ screen purposes, instructions are few and simple. 35 

B6b. Instructions are clear; They indicate whether multiple responses are allowed. 30 

D5. If reference is made to a previous item, that item appears on the same page or on 

the facing page. 30 

D7d. Within a topic/content area, the items progress from objective to subjective. 30 

D8. Items that require recall are organized by logical time sequence. 25 



was there a checklist with column headings (ranging from outstanding to needs improvement) repeated 
at the top of the second page. One questionnaire used the Likert Strongly Agree/Strongly Disagree 
format with graduates expected to circle their response (SA to SD) for each item. One questionnaire 
instructed graduates to rate various aspects of their program (grouped into dusters of items) using a 
scale from 0 to 4 that was defined only once on the first page. The final questionnaire directed 
graduates to rate aspects of their preparation by circling a number from 0 to 5 for each item, but the 
descriptions of the ratings ascribed to the numbers appeared only on the first page. 

Item D5 (30%), for example, refers to "facing pages," but responses to items on the Procedures 
Used in Moat Recent Follow-up Surveys indicated that only two of the 20 questionnaires included them. 
Item E4b (35%) refers to items used for skip/filter/screen purposes. Of the eight sample instruments 
that were submitted as examples, only two included any type of skip or filter items. None of the eight 
sample questionnaires included ranking items, thus item C13 (35%) was irrelevant in relation to them. 

Some of the follow-up researchers did, however, think some items not related to their surveys 
were important For example, item F3e concerning items utilizing agree/disagree response format was 
considered important by 75% of the study participants, but only one of the eight sample instruments 
contained agree/disagree items. 

Summary 

There are commonalities between the design consideration considered important by follow-up 
survey researchers and by the experts and experienced general survey researchers in the previous 
research. There also are differences in the procedures used and questionnaire characteristics in the 
surveys conducted with the general and specific populations. There were differences among the follow- 
up surveys in dimensions such as size of sample/ population surveyed and questionnaire length, While 
not addressed specifically in this study, it has been noted that follow-up surveys also may serve varied 
purposes (occupational information, future plans, program evaluation), thus some of the surveys may 
have included types of questions not included in others (Le. attitude/opinion questions). 

Some lack of agreement between successful follow-up survey researchers and the group of 
experts and experienced general survey researchers may arise from differences in the nature of the 
survey and types of questions asked. Since the researchers in this study were successful in their surveys, 
it would appear that procedural differences may be warranted and may not be detrimental to a survey 
if the researcher knows, understands, and takes into consideration the target population as well as the 
circumstances related to the specific survey effort. 
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Appendix 

Percentage of Respondents Identifying Questionnaire Design Considerations 

as Important 

(N-20) 



Percentage 
Rating Item 



Item Important 

A. General Appearance 

1 . The title of the study/questionnaire is likely to appeal to the survey 

population 60 

2. Instrument looks easy to complete 100 

3. Margins are adequate; instrument doesn't look crowded 65 

4. Paper is white or light-colored with dark ink. 75 

5. Printing does not bleed through paper 80 

6. Type is clear and legible. 95 

7. Size and style of type used for headings is consistent through the 

instrument. Consistency is also evident for ifans and response options. 65 

8. There are not too many variations in size and style of type. 45 

9. The front page (or cover) contains: 

a. the study /instrument title, prominently displayed. 55 

b. general directions. 85 

c. the name of the sponsor. 73 

d. the address of the sponsor. 55 

10. For a multi-page questionnaire, the back page does not contain items 

but may be used for comments. 25 

1 1 . Appreciation for completing the instrument is expressed. 70 

B. Instructions 

1 . General instructions that apply to the entire instrument are provided 

at the beginning of the instrument. 90 

2. Instructions are brief. 100 

3. Instructions are visually different from the body of the instrument 

(e.g., in size and/or style of type). 55 

4. The tone of the directions is polite (e.g., 'please"). 75 

5. If items appear on both sides of the page, an indication is given that the 

instrument continues on the reverse side (e.g., "Please turn over"). 55 

6. Instructions are dear. 

a. They specify when to put a check mark and when to write in a response. 80 

b. They indicate whether multiple responses are allowed. 30 

c. They provide guidance for expected length of open-ended responses 30 

C. Item Format 

1 . Items are numbered with Arabic numerals. 55 

2. If necessar/, either sublettering (e.g., 4a, 4b, 4c) or numbering by 
sections (i.e., starting each section with item 1) is used to limit the 

apparent number of items. 25 

3. Each item anc its response options are on the same page. 80 

4. Statements or questions, rather than phrases, are used in collecting 
demographic information (e.g., "How old were you on your last birthday?'' 

instead of "Age"). 35 
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Percentage 
Rating Item 



Item Important 



C Item Formal (continued) 

5. If an item stem requires two or more lines, the second and subsequent 

lines are indented. 20 

6. The respondent is asked to drcle or underline responses already 

presented rather than write them on a blank. 65 

7. When response options are provided (including, if appropriate, a 
response option of "other"), each response option has either a numeric 

or alphabetic code beside it 40 
S. Response options are arranged vertically (or in columns if several 

consecutive items use the same response options). 55 
9. Response options are close to the item stem. 60 

10. The space for responding to items is on the same side of the page 

throughout the instrument. 60 

11. There is adequate space for responding. 95 

12. Open-ended items are used sparingly. 75 

13. When ranking, the number of items to be ranked is limited (e.g., three 

best and three worst). 35 

14. For checklists: 

a. If long, a line is skipped after every three to six items. 15 

b. Column headings are carried over from one page to another 35 

c. Column headings are presented parallel, rather than j^rpendicular, 

to the item stem. 10 

D. Order of Items 

1. The initial items are: 

a. clearly connected to the stated purpose of the survey. 85 

b. applicable to all members of tine survey population 80 

c. easy 65 

d. nonthreatening 70 

e. interesting 35 

2. If there are any sensitive or difficult items, they appear in the middle 

or near the end of the instrument, but not at the very end. 35 

3. Open-ended items appear last. 55 

4. Classification or demographic information is solicited at the end of the 

instrument unless iieeded for screening purposes. 5 

5. If reference is made to a previous item, that item appears on the 

same page or on the facing page. 30 

6. Items with similar content are grouped together; within each content 

group, items with the same response format are presented together. 45 

7. Within a topic/content area, the items progress from: 

a. general to specific. 35 

b. most familiar to least familiar. 20 

c. least objectionable to most objectionable. 15 

d. objective to subjective 30 

8. Items that require recall are organized by logical time sequence. 25 
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Percentage 
Rating Item 

^ ism Important 

E. Choice of Items 

1. The respondent is able to provide answers to the questions in the instrument. 85 

2. Each item seeks fust one piece of information. 90 

3. All items are essential and relevant to the purposes of the survey. 95 

4. For items used for skip/filter/screen purposes: 

a. The use of this type of item is justified. 20 

b. Instructions are few and simple. 35 

c. Instructions appear immediately after the response options. 25 

d. Items pertaining to only some of the respondents are indented beneath 

the filter question. 10 

F. Wording 

1. The choice of words is appropriate to the literacy level of the survey 

population. 80 

2. Both sides of an issue (or neither side) are included in the item stem. 30 

3. Items are simple, direct, and unambiguous. They have /: 

a. no Jargon, technical terms, or uncommon abbreviations. 75 

b. no loaded'* items (that use emotionally colored words). 75 

c. no assumption of an existing state of affairs (e.g., "Do you still..."). 75 

d. no double negatives in items and/or response options. 80 

e. no negatively worded items coupled with agree/disagree response format 75 

f. no qualifying clauses, especially at end of stem. 50 

g. no "giveaway* words (e.g., "all"). 65 

h. no inexact words or phrases (e.g., "any," "most," "several,*' "usually," 

"often," "regularly/ "much the same'). 50 

i. no vague terminology (e.g., "the country," "just," "fair," "you"). 60 
)• not used the word "questionnaire" or "checklist" in heading or title. 40 

G Choice of Response Options 

1. Response options: 

a. exhaust all possibilities or include "other," "undecided," or "neutral" 

category. 40 

b. are mutually exclusive. 40 

c. include a "don't know" option. 30 

d. do not contain more than one alternative that could be correct 

unless multiple responses are allowed (Le., "check all that apply"). 40 

e. include both side, of the issue among the choices. 35 

f . are brief. 90 

g. are appropriate for the item 80 

2. Items with Likert-rvpe response options: 

a. have an appropriately labeled midpoint. 60 

b. use a balanced scale. 65 

3. Sensitive informant,. ■ .'e.g., age, salary) is collected using ranges for 

response options. 45 
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